DOCUMENT RESUME 



ED 456 790 



HE 034 350 



AUTHOR 

TITLE 



PUB DATE 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Wisan, Gail; Nazma, Shirin; Pscherer, Charles P., Jr. 
Comparing Online and Face- to- Face Instruction at a Large 
Virtual University: Data and Issues in the Measurement of 
Quality. AIR 2001 Annual Forum Paper. 

2001-06-05 

28p . ; Paper presented at the Annual Meeting of the 
Association for Institutional Research (41st, Long Beach, 

CA, June 3-6, 2001). 

Reports - Research (143) -- Speeches/Meeting Papers (150) -- 

Tests/Questionnaires (160) 

MF01/PC02 Plus Postage. 

*Distance Education; *Educational Quality; Graduate Surveys; 
Higher Education; * Instructional Effectiveness; Measurement 
Techniques; *Online Systems; *Student Attitudes; 
*Undergraduate Students 
Virtual Universities 



ABSTRACT 



The fiscal year 2000 Alumni Survey at a large, substantially 
online university was used to assess undergraduate students' satisfaction 
with both online and face-to-face academic quality and student services. 
Students who had taken online classes evaluated both their online and 
face-to-face classes. In addition, students who took only face-to-face 
classes evaluated them. Responses were received from 567 graduates, 
representing a 31% response rate. Differences between the online and 
face-to-face students' ratings of face-to-face academic quality raised 
questions about the significance of context for self-reported data. Several 
of the differences in rating actually favored face-to-face among students who 
took only one online course. However, when students took four or more online 
courses, they rated their online classes as superior in their satisfaction 
with such key academic quality indicators as "develop critical thinking" and 
"rigor and scholarship." Other measurement issues related to whether online 
and face-to-face classes should be compared are discussed. One appendix 
contains a survey overview, and the other contains the survey. (Contains 6 
tables and 17 references.) (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



>34350 



W 

Comparing Online and Face-to-Face Instruction at a 
Large Virtual University: Data and Issues in the 
Measurement of Quality 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

~ P-V(/rA . 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Gail Wisan, Ph.D. 
Director 



U S DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Shirin Nazma, Ph.D. 

Senior Research Associate 

Charles P. “Pat” Pscherer, Jr. 
Research Associate 



Office of Institutional Accountability, Planning and Research 
University of Maryland University College 
3501 University Boulevard East 
Adelphi, MD 20783-8010 



Paper presented at: 41 s ' Forum of 

The Association for Institutional Research 
Long Beach, California, 

June 5, 2001 



ERIC 



2 



BEST COPY AVAILABLE 



Comparing Online and Face-to-Face Instruction at a 

Large Virtual University: Data and Issues in the 
Measurement of Quality 



Abstract 

The FY 2000 Alumni Survey at a large, substantially online university was used to assess 
undergraduate students' satisfaction with both online and face-to-face academic quality 
and student services. Students who had taken online classes evaluated both their online 
and face-to-face classes. Additionally, students who took only face-to-face classes 
evaluated them. Differences between the online and face-to-face students' ratings of 
face-to-face academic quality raised questions about the significance of context for self 
reported data. Other measurement issues relating to whether online and face-to-face 
classes should be compared are discussed. 



Comparing Online and Face-to-Face Instruction at a 

Large Virtual University: Data and Issues in the 
Measurement of Quality 

Introduction 

Outcome Assessment 

Interest in assessment at institutions of higher education, while important for decades, 
has recently surged. For example, the Middle States Commission on Higher Education 
recently offered a paper on the Characteristics of Excellence: Standards for 
Accreditation (2001), which concludes with a section on educational effectiveness. In the 
chapter on institutional assessment, the Middle States Commission notes that outcomes 
assessment has two purposes: "accountability" and "effectiveness". The Commission 
goes on to say: 

External forces and internal priorities require institutions of higher education to 
demonstrate their effectiveness and efficiency to students and the broader public 
as well. (pg. 43) 

Effectiveness can be explored at both the micro level (e.g., student learning) and at 
the macro level (e.g., institutional effectiveness). One tool for studying institutional 
effectiveness and quality is the university's alumni survey. In the Middle States 
Commission’s Characteristics of Excellence, they list a number of such tools and 
documents to use in the analysis and documentation of an institution's excellence. 
"Analysis of student satisfaction survey results" is listed as an assessment tool to help an 
institution perform outcome assessment and evaluate its overall effectiveness, (pp. 43-44) 
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Growth in Alternative Delivery Formats and Non-Traditional Institutions 



The surge in interest in assessment at higher education institutions may, at least in 
part, flow from the surge in interest and growth in alternative education delivery formats 
and non-traditional institutions. For example, much publicity heralded the birth of the 
non-traditional Western Governor's University, and it was given a special position in the 
Department of Education's Distance Education Demonstration Project. Moreover, 
students have been "voting" for the new online delivery formats with their registrations. 
Online enrollment growth at the University of Maryland University College has been in 
double and triple digits in recent years and has grown at a fifty percent rate in FY 2001 
over a large base in FY 2000. UMUC's worldwide enrollments/duplicated registrations 
have reached 60,000 worldwide. Other non-traditional institutions (e.g., for-profit degree- 
granting) have also moved into the online arena. 

Specific Research Question 

This paper focuses on comparing student satisfaction with online and face-to-face 
instruction at a large university that currently has about half of its undergraduate 
enrollments/registrations online. Because online students are not segregated into special 
continuing education schools or specifically-online colleges, many students are actually 
"mixed," (i.e., they can take both online and face to face classes at the same time or over 
the course of their academic career). Students who take online classes may take only one 
class online or, at the other end of the spectrum, take all of the classes they need for their 
bachelor's degree online. Thus, it is possible to compare online students', who could be 
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described as "mixed" format students, evaluations of both their online classes and their 



face-to-face classes in a series of questions in the Bachelor's Alumni Survey. Rather than 
simply comparing students’ assessment of their online and face-to-face classes, we 
analyzed the results in terms of the number of online classes the students had taken. 
Three levels were examined: students who had taken only one online course; students 
who had taken two or three online courses; and students who had taken four or more 
online courses. 



Research Design 

The FY2000 Alumni Survey, conducted one year after graduation, was used to assess 
FY 1999 bachelor's degree recipients' satisfaction with both online and face-to-face 
academic quality indicators as well as satisfaction with the quality of student services. 

We compared students taking online and face-to-face classes ratings of both their face-to- 
face classes and their online classes. 

In a university that provides face-to-face, online and some other types of distance 
classes and where students can enroll in different delivery formats during the same 
semester or over time, students may have a different view of distance and face-to-face 
classes at different times during the course of their enrollment. After the degree is 
awarded, students have perceptions about their online and face-to-face education. The 
alumni survey provides a good vehicle to measure students' perceptions and satisfaction 
with their online and face-to-face educational experience at their alma mater. Analysis of 
students' self-reported data on their satisfaction with online and face-to-face classes is an 
important assessment tool for a university's self-study. The matched pair t-test was used 
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to assess whether there was significant difference between how alumni rated their face- 
to-face classes and their online classes. 

In “Quality On the Line: Benchmarks for Success in Internet-Based Distance 
Education” (1999), the report prepared with the support of the National Education 
Association and Blackboard, Inc., Phipps and Merisotis recommend as a benchmark that: 
The program's educational effectiveness and teaching/leaming process [be] 
assessed through an evaluation process that uses several methods and applies 
specific standards, (pg. 16) 

The alumni survey provides one method of assessing the effectiveness and quality of 
the face-to-face and online teaching/leaming process. Other assessment processes are 
normal testing, writing and portfolio assignments, and teacher/course evaluations. 



Methodology 

The main objective of this study was to compare students' level of satisfaction with 
face-to-face and online instruction. Furthermore, we analyzed the relationship between 
student satisfaction and the number of online classes in which a student had enrolled. 

Data Collection and Indicators 

All bachelor’s level students who graduated stateside 1 (n=l,802) during FY 1999 
(i.e., summer 1998, fall 1998, and spring 1999) were sent an Alumni Survey in the spring 
of 2000. There were 567 respondents, representing a 31% response rate. (See Appendix 
A for a demographic comparison of the respondents and the FY 1999 graduates.) The 
survey contained fourteen items measuring the excellence or quality of the academic 
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program (see Table 1) and nine items measuring the quality of the delivery of student 
services and support (see Table 2). Alumni were asked to rate each item on a five-point 
Likert scale where 1 equaled “Very Dissatisfied” and five equaled “Very Satisfied.” 
Higher mean scores thus represented higher levels of satisfaction with academic quality 
and student services. 

Among the total 567 bachelor's level respondents, 330 alumni rated both their face-to- 
face and online courses. An additional 232 alumni took only face-to-face courses and 
thus rated only face-to-face classes on the twenty-three quality and effectiveness 
indicators. (Note: Five respondents did not respond to these questions about quality and 
effectiveness; i.e., item nonresponse, thus explaining the discrepancy between the 567 
total respondents and the 562 respondents used for analysis.) Alumni were asked to 
indicate their satisfaction with their alma mater's performance on each of the items. See 
Appendix B for an example of the original questions. 

Comparing Means: Matched Pairs T-Test 

For each of the twenty-three quality/effectiveness indicators, only alumni who 
evaluated both online and face-to-face classes were used. For each assessment item, the 
respondent's mean satisfaction ratings for online classes was calculated separately from 
their mean satisfaction ratings for their face-to-face classes. In order to evaluate the 
research objective as to whether alumni reported any significant difference between their 
online and face-to-face ratings, a matched t-test was calculated. For each item, the mean 
of the differences between respondent's ratings of online and face-to-face classes was 
calculated. To calculate the t-statistics for matched samples, mean of the differences 



between online and face-to-face ratings is divided by the standard error. The p value of 
the test was determined. A decision was then made for each item as to whether the 
difference between the online and face-to-face classes was statistically significant and 
thus whether alumni rated online and face-to-face classes differently on that quality and 
effectiveness indicator. 

Hypotheses and tests 

Ho: true mean difference is zero (i.e., d = 0) 

H a : true mean difference is not equal to zero (i.e., d ^ 0). 

The test statistic 

T - 0 

s(d ) 

In the above formula, 

d = the mean of the differences between each respondents' satisfaction ratings for online 
classes and face-to-face classes for each quality indicator. 

s (d) is the standard error of the mean differences . 

Where s (d) = s d / square root of n .. 

Where s d = square root (£(di - d ) 2 )/ n-1) 



Analysis of Level of Online Enrollment 

To evaluate the second stated objective, the students who took the online courses 
were classified into three groups: (a) alumni who had taken only one online course, (b) 
alumni who had taken two or three online courses, and (c) alumni who had taken four or 
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more online courses. Means for each quality indicator were calculated for each category 
of online enrollment. The online mean ratings of students within each category were 
compared to their face-to-face ratings and differences were calculated. The matched t-test 
for differences between the mean levels of satisfaction for each quality indicator was 
calculated for both the online and face-to-face ratings of instruction and student services. 
Decisions were made regarding whether the differences were statistically significant at 
the .05 level. 

Survey Items 

In addition to many standard questions, students were presented with a set of 
questions that asked them about their ratings of specific criteria of academic quality. 
Many of these questions are also included in the U.S. Department of Education's Distance 
Education Demonstration Project Distance Education Survey. 



Results and Discussion 

Rating Online and face-to-face Classes: 

Matched Pair T-Test for Significant Differences 

Table 1 presents the results of a matched-pairs t-test of online students rating online 

and face-to-face courses. Results are presented for the mean for each item for the 

alumni's online classes and the alumni's face-to-face classes. The mean differences 

between respondents’ ratings of their online and face-to-face classes are shown. The 

matched sample t-test for the difference of means is calculated and the p value is 

determined. The .05 level was used as the cut off level for significance. 
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Table 1 lists the mean of the differences in responses between online and face-to-face 
courses regarding the levels of satisfaction for each of the fourteen criteria. Out of these 
fourteen indicators for quality in academic programs, we found that the mean of the 
differences were statistically significant at p <= .05 for twelve of the indicators. Among 
the twelve significant indicators, the mean of the differences for ten criteria were negative 
and the negative values indicate that the levels of satisfaction on face-to-face instruction 
were higher, as compared to the levels of satisfaction on online instruction. On two 
criteria, 'flexibility of courses' and 'course availability', the levels of satisfaction for online 
instruction were significantly higher as compared to the ratings of face-to-face 
instruction. No statistically significant differences in the level of satisfaction were found 
in 'develop critical thinking' or 'rigor and scholarship' in Table 1. Table 1 includes all 
students who took 1 or more online class and who also evaluated their face-to-face 
classes. 

Regarding services and support, the mean for the differences of 'availability of career 
planning services’ (p= .0324) was the only indicator that was statistically significant, and 
alumni rated face-to-face delivery higher than online. No significant differences were 
found on any other service dimensions, indicating that the students were equally satisfied 
with the online delivery they received. 
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Table 1 



Matched t-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of Their 
Face-to-Face and Online Classes' Academic Quality 



Academic Indicator 


n 


Mean 


Mean 

Difference 


Standard 

Error 


P-value 


Face-to- 

Face 


Online 


Content of curriculum 


236 


4.254 


4.089 


-0.165 


0.048 


0.0008* 


Quality of education 


240 


4.279 


4.009 


-0.270 


0.057 


<0.0001* 


Timeliness of feedback 


237 


4.248 


3.856 


-0.392 


0.062 


<0.0001* 


Testing and grading 


238 


4.239 


3.975 


-0.264 


0.058 


<0.0001* 


Faculty knowledge 


236 


4.305 


4.132 


-0.173 


0.055 


0.0019* 


Faculty teaching skills 


238 


4.134 


3.853 


-0.281 


0.068 


<0.0001* 


Academic advising 


207 


3.748 


3.478 


-0.270 


0.061 


<0.0001* 


Develop critical thinking 


235 


4.055 


4.004 


-0.051 


0.054 


0.3469 


Rigor and scholarship 


226 


3.911 


3.854 


-0.057 


0.051 


0.2641 


Student-faculty 

interaction 


236 


4.139 


3.652 


-0.487 


0.069 


<0.0001* 


Student-student 

interaction 


232 


4.103 


3.457 


-0.646 


0.070 


<0.0001* 


Flexibility of courses 


236 


3.995 


4.245 


0.250 


0.067 


0.0003* 


Course availability 


235 


3.829 


4.126 


0.297 


0.064 


<0.0001* 


Relevance to career 


237 


4.202 


4.059 


-0.143 


0.049 


0.0041* 
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Table 2 



Matched t-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of 
Face-to-Face and Online Services and Support 



Services and Support 


N 


Mean 


Mean 

Difference 


Standard 

Error 


P-value 


Face-to- 

Face 


Online 


Availability of computer 
laboratories 


154 


4.045 


3.948 


-0.097 


0.055 


0.0833 


Availability of tutoring services 


105 


3.457 


3.391 


-0.066 


0.054 


0.2247 


Availability of career planning 
services 


113 


3.230 


3.142 


-0.088 


0.040 


0.0324* 


Availability of computer 
training/information literacy 


146 


3.671 


3.603 


-0.068 


0.044 


0.1232 


Availability of library and other 
learning material 


179 


4.039 


3.967 


-0.072 


0.038 


0.0577 


Ease of registration 


220 


4.581 


4.612 


0.031 


0.028 


0.2633 


Availability of technical 
assistance 


173 


3.913 


3.890 


-0.023 


0.057 


0.6904 


Co-op services and support 


92 


3.847 


3.771 


-0.076 


0.038 


0.0517 


Excel services and support 


83 


3.831 


3.735 


-0.096 


0.055 


0.0881 
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Table 3 and Table 4 present the results of matched-pairs t-testfor three 
levels of enrollment in online courses. 

Students taking only 1 online course 

Table 3 lists the mean of the differences in responses between online and face-to-face 
courses regarding the levels of satisfaction for each of the fourteen criteria. Out of 
fourteen criteria regarding academic quality, the mean of the differences were statistically 
significant at p<=.05 for twelve criteria. The mean of the differences for all twelve 
criteria were negative and the negative values indicate that the levels of satisfaction on 
face-to-face instruction were higher, as compared to the levels of satisfaction on online 
instruction. No statistically significant difference in the level of satisfaction was found in 
'flexibility of courses' or 'course availability' criteria. 

Regarding services and support (Table 4), the mean of the differences of 'availability 
of career planning services' (p=.0486) was the only criteria that was found statistically 
significant, and it was scored higher on face-to-face than online. No significant 
difference was found on any other criteria. 

Students taking 2-3 online courses 

Out of fourteen criteria regarding education, the mean of the differences were 
statistically significant at p<=.05 for eleven criteria. Among the eleven significant 
criteria, the mean of the differences for ten criteria were negative and the negative values 
indicate that the levels of satisfaction on face-to-face instruction were higher, as 
compared to the levels of satisfaction on online instruction. On the criteria: 'course 
availability', the level of satisfaction for online instruction was significantly higher 
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compared to the level for face-to-face instruction. No statistically significant difference 
in the level of satisfaction was found in 'develop critical thinking', 'rigor and scholarship' 
or 'flexibility of courses criteria.' 

In Table 4, regarding services and support, the mean of the differences of three 
criteria 'availability of computer laboratories' (p=.0219), 'availability of tutoring services' 
(p=.0188), and 'availability of library and other learning materials' (p=.01 11) was found 
statistically significant, and they were scored higher on face-to-face than online. No 
significant difference was found on the other criteria. 

Students taking 4 or more online courses 

Out of fourteen criteria regarding academic quality, the mean of the differences were 
statistically significant at p<=.05 for five criteria. Among the five significant criteria, the 
mean of the differences for 4 criteria were positive and the positive values indicate that 
the levels of satisfaction on online instruction were higher, as compared to the levels of 
satisfaction on Face-to-Face instruction. The four significant criteria were: 'develop 
critical thinking' (p=.0023), 'rigor and scholarship' (p=. 0142), 'flexibility of courses' 
(p=<.0001) and 'course availability' (p=.0009). On the criteria 'student-student interaction' 
(p=<0.0009)', the levels of satisfaction on face-to-face instruction were significantly 
higher compared to the levels on online instruction. No statistically significant difference 
in level of satisfaction was found on any other academic or student service delivery 
dimension. 
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Table 3 



Matched t-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of Their 
Face-to-Face and Online Classes' Academic Quality 

Three Levels Based Upon Number of Online Course Enrollments 



Academic Indicator 


1 Online Course 


2-3 Online Courses 


4 or More Online Courses 


Mean 

differences (n) 


P value 1 


Mean 

differences (n) 


P value 1 


Mean 

differences (n) 


P value 


Content of curriculum 


-0.372 (43) 


0.0062* 


-0.259(81) 


<.0001* 


0.121 (82) 


0.1235 


Quality of education 


-0.590 (44) 


0.0005* 


-0.428 (84) 


<.0001* 


0.073 (82) 


0.3577 


Timeliness of feedback 


-0.720 (43) 


0.0006* 


-0.385 (83) 


<.0001* 


-0.172 (81) 


0.0657 


Testing and grading 


-0.558 (43) 


0.0021* 


-0.285 (84) 


0.0010* 


-0.073 (82) 


0.4420 


Faculty knowledge 


-0.441 (43) 


0.0087* 


-0.301 (83) 


0.0002* 


0.060 (82) 


0.5320 


Faculty teaching skills 


-0.651 (43) 


0.0017* 


-0.361 (83) 


0.0007* 


0.060 (82) 


0.5505 


Academic advising 


-0.61 1 (36) 


0.0031* 


-0.333 (69) 


0.0009* 


-0.065 (76) 


0.4787 


Develop critical thinking 


-0.477 (44) 


0.0057* 


-0.148 (81) 


0.0573 


0.271 (81) 


0.0023* 


Rigor and scholarship 


-0.512(41) 


0.0014* 


-0.116(77) 


0.1290 


0.212 (80) 


0.0142* 


Student-faculty 

interaction 


-0.813(43) 


0.0002* 


-0.686 (83) 


<.0001* 


-0.148 (81) 


0.1527 


Student-student 

interaction 


-0.930 (43) 


<.0001* 


-0.756 (82) 


<.0001* 


-0.379 (79) 


0.0009* 


Flexibility of courses 


-0.113(44) 


0.5141 


0.172(81) 


0.1274 


0.543 (81) 


<.0001* 


Course availability 


0.000 (43) 


1.0000 


0.308 (81) 


0.0029* 


0.419(81) 


0.0009* 


Relevance to career 


-0.444 (45) 


0.0061* 


-0.268 (82) 


0.0007* 


0.074 (81) 


0.1093 
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Table 4 



Matched t-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of 
Face-to-Face and Online Services and Support 

Three Levels Based Upon Number of Online Course Enrollments 



Services and Support 


1 Online Course 


2-3 Online Courses 


4 or More Online Courses 


Mean 

differences (n) 


P value 1 


Mean 

differences (n) 


P value 1 


Mean 

differences (n) 


P value 


Availability of computer 
laboratories 


-0.103 (29) 


0.3256 


-0.222 (54) 


0.0219* 


0.054 (55) 


0.6061 


Availability of tutoring 
services 


-0.266 (15) 


0.2170 


-0.210(38) 


0.0188* 


0.119(42) 


0.1333 


Availability of career planning 
services 


-0.411 (17) 


0.0486* 


-0.052 (38) 


0.4215 


-0.021(47) 


0.5693 


Availability of computer 
training/information literacy 


-0.260 (23) 


0.0557 


-0.125(48) 


0.1825 


0.070 (57) 


0.2087 


Availability of library and 
other learning material 


-0.033 (30) 


0.7450 


-0.175 (57) 


0.0111* 


0.000(68) 


1.000 


Ease of registration 


-0.026 (38) 


0.7108 


-0.013(74) 


0.8102 


0.075 (79) 


0.1093 


Availability of technical 
assistance 


-0.125 (32) 


0.4882 


-0.145 (62) 


0.1511 


0.155 (58) 


0.0599 


Co-op services and support 


-0.133(15) 


0.3343 


-0.138 (36) 


0.0576 


0.033 (30) 


0.3256 


Excel services and support 


-0.200(10) 


0.3434 


-0.156 (32) 


0.1691 


0.034 (29) 


0.5728 
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Rating Face-to-Face Classes - Methodological Implications: 

Students taking Online classes and Students taking only Face-to-Face Classes 

The rating of face-to-face classes can be compared for students taking online classes 
(mixed students) and students taking only face-to-face classes. This comparison is not 
typically done since the focus is usually upon the online classes or comparing online and 
face-to-face classes. This approach is valuable as a tool for identifying what we have 
called the "Dartmouth Effect." When theoretical discussions about quality and standards 
attempt to compare online and face-to-face classes, the online classes are compared to a 
theoretical face-to-face ideal. In other words, the online class is compared not to the 
reality of the large lecture hall class of 100 to 800 students, or to a small class with an 
instructor who mainly lectures, but rather to an idealized, small face-to-face class with 
substantial discussion. In other words, a stereotypical Dartmouth-style classroom setting. 
We hypothesize that the same idealization occurs when alumni students who have taken 
both online and face-to-face classes remember and rate their face-to-face classes in the 
Alumni Survey. We hypothesized that students who were rating their online experience 
would tend to inflate the scores of their face-to-face classes and would have higher 
ratings on all quality and service ratings items for their face-to-face classes than the 
students just rating face-to-face classes. 

Ho: p .1 - p .2 < 0 
Ha: m - p .2 >0 

p,i =mean ratings of face-to-face classes by alumni who have taken online classes. 

\i 2 = mean ratings of face-to-face classes by alumni who have taken only face-to-face 
classes. 
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We test these hypotheses by using the z-test statistic. We use .05 as the level of 
significance required to reject the null hypothesis. 

Items on which online students rated face-to-face classes significantly higher than 
face-to-face students were: ‘timeliness of feedback’, ‘rigor and scholarship’, ‘student-faculty 
interaction’, and ‘student-student interaction.’ 

Items on which online students rated face-to-face classes higher but which only 
reached the .10 level of significance (i.e., not high enough to reject the null hypothesis) 
were: ‘academic advising’ and develop critical thinking.’ 

For the convenience items, ‘flexibility of courses’ and ‘course availability’, we 
hypothesize that online students will rate face-to-face classes lower than alumni who 
have taken only face-to-face classes. 

Ho: |ii - |X 2 > 0 
Ha: p.i - p, 2 < 0 

Alumni who had taken online classes rated face-to-face classes significantly lower 
than face-to-face students in terms of ‘flexibility of course’. Table 5 lists all of the quality 
items with the mean ratings of face-to-face classes by the two comparison groups: 1) 
alumni who had taken online and face-to-face classes and rated them both, and 2) alumni 
who had taken only face-to-face classes and thus rated only face-to-face classes. See 
Table 5 for the p-value for the one-tailed test. See Table 6 for a list of all the services and 
support items with the mean ratings of face-to-face classes by the above two comparison 
groups. 
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For the services and support items, the hypotheses were: 

Ho: |ii - |i 2 ^ 0 
Ha: |ii - |i 2 > 0 

(|ii =mean ratings of face-to-face classes by alumni who have taken online classes. 
p .2 = mean ratings of face-to-face classes by alumni who have taken only face-to-face 
classes. 

We tested these hypotheses by using the z-test statistic. We used .05 as the level of 
significance required to reject the null hypothesis. See Table 6 for the p-value for the 
one-tailed test. Items on which online students rated face-to-face services and support 
significantly higher than face-to-face students were: ‘availability of technical assistance’, 
‘co-op services and support’, and ‘Excel services and support’. 

The higher evaluation of face-to-face classes by the online alumni as compared with 
face-to-face students provides some preliminary evidence that online students may 
unintentionally inflate their ratings of face-to-face classes. Just as when face-to-face 
classes are compared in the abstract to online classes, there is a tendency to idealize face- 
to-face classes, the same effect may be at work in general assessments such as in an 
alumni survey. Face-to-Face classes ratings may benefit among online students from the 
"Dartmouth Effect," whereby face-to-face classes are idealized to the stereotyped small, 
highly interactive discussion class. If such an effect were not at work, students rating 
online classes evaluations of their face-to-face class should not differ significantly from 
face-to-face students rating of their face-to-face classes. While these results are 
preliminary and need to be tested further in the future, they do raise methodological 
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questions about the problems of rating face-to-face classes and online classes in the same 
context. 
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Table 5 



One-tailed z-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of 
Face-to-Face Classes’ Academic Quality 



Academic Indicator 


Online Stu< 


ents Evaluati 


ng F-t-F 


F-t-F Students Evaluating F-t-F 


P(Z<=z) 
One tail 


Mean 


Standard 

Error 


n 


Mean 


Standard 

Error 


n 


Content of curriculum 


4.22 


0.042 


283 


4.2 


0.046 


215 


0.387 


Quality of education 


4.21 


0.048 


285 


4.22 


0.537 


217 


0.479 


Timeliness of feedback 


4.21 


0.043 


281 


4.09 


0.049 


215 


0.045* 


Testing and grading 


4.19 


0.044 


284 


4.13 


0.049 


217 


0.182 


Faculty knowledge 


4.26 


0.047 


283 


4.20 


0.055 


216 


0.215 


Faculty teaching skills 


4.10 


0.055 


283 


4.06 


0.055 


217 


0.335 


Academic advising 


3.78 


0.066 


271 


3.63 


0.076 


206 


0.067 


Develop critical thinking 


4.01 


0.053 


281 


3.88 


0.058 


217 


0.051 


Rigor and scholarship 


3.86 


0.055 


271 


3.67 


0.066 


208 


0.014* 


Student-faculty 

interaction 


4.06 


0.054 


280 


3.87 


0.057 


216 


0.008* 


Student-student 

interaction 


4.06 


0.051 


282 


3.84 


0.059 


213 


0.002* 


Relevance to career 


4.15 


0.048 


283 


4.09 


0.065 


216 


0.205 



Items for which Students Rated Online Classes’ Academic Quality 
As Higher than that for Face-to-Face 



Academic Indicator 


Online Students Evaluati 


ing F-t-F 


F-t-F Students Evaluating F-t-F 


P(Z<=z) 
One tail 


Mean 


Standard 

Error 


n 


Mean 


Standard 

Error 


n 


Flexibility of courses 


4.01 


0.050 


282 


4.14 


0.055 


217 


0.041* 


Course availability 


3.84 


0.058 


282 


3.94 


0.069 


217 


0.124 
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Table 6 



One-tailed z-test: Comparison of Mean Satisfaction 

Alumni Students' Ratings of 
Face-to-Face Services and Support 



Services and Support 


Online Students Evaluating F-t-F 


F-t-F Students Evaluating F-t-F 


P(Z<=z) 
One tail 


Mean 


Standard 

Error 


N 


Mean 


Standard 

Error 


N 


Availability of computer 
laboratories 


3.99 


0.059 


225 


3.96 


0.070 


183 


0.379 


Availability of tutoring 
services 


3.42 


0.067 


149 


3.30 


0.082 


132 


0.130 


Availability of career 
planning services 


3.27 


0.080 


162 


3.14 


0.087 


137 


0.143 


Availability of computer 
training/information 
literacy 


3.56 


0.064 


194 


3.48 


0.082 


146 


0.235 


Availability of library 
and other learning 
material 


3.97 


0.063 


235 


3.95 


0.069 


185 


0.443 


Ease of registration 


4.59 


0.038 


271 


4.61 


0.047 


212 


0.326 


Availability of technical 
assistance 


3.85 


0.059 


214 


3.48 


0.073 


164 


0.00003* 


Co-op services and 
support 


3.81 


0.095 


130 


3.27 


0.102 


106 


0.00005* 


Excel services and 
support 


3.87 


0.092 


132 ' 


3.48 


0.100 


112 


0.002* 
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Implications 



The self-reported alumni satisfaction survey data has implications for both the 
assessment of quality in education and the importance of intervening variables in the 
assessment of quality of online education. Several of the differences in rating actually 
favored face-to-face among students who took only one online course. However, w hen 
looking at students who took four or more online courses, they rated their online classes 
as superior in their satisfaction with such key academic quality indicators as : "develop 
critical thinking" and "rigor and scholarship. " 



End Notes 

1. University of Maryland University College operates three divisions globally: 1) a 
European division headquartered in Heidelberg, Germany, 2) an Asian division, 
headquartered in Tokyo, Japan, and 3) the Stateside division, operating locations in the 
Maryland-DC-Northem Virginia area, headquartered in Adelphi, Maryland. The 2000 
Alumni Survey of 1999 Degree Recipients was conducted using a sampling of students 
receiving their degrees after completing programs administered by the Stateside division. 
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Appendix A 

Survey Overview 

Mailed: summer 2000 
Number mailed to undergraduates: 1,802 
Number of respondents: 567 
Response rate: 31% 

Comparison of Alumni Survey Respondents to the 
FY1999* Bachelor's Degree Recipient Population 



Demographic 


Alumni Survey 
Respondents 


AH Bachelor’s 
Degree Recipients 


Gender 


Male 


42% 


47% 


Female 


58% 


53% 


Ethnicity 


African-American 


22% 


24% 


American Indian 


0% 


<1% 


Asian 


5% 


7% 


Hispanic 


4% 


4% 


White 


68% 


63% 


Unknown 


1% 


2% 



*FY 1999 degree recipients received their degrees in August 1998, December 1998, and June 1999. 



Appendix B 

Questionnaire for Recent Graduates 



Please use the following rating scale to answer Question 41 and Question 42. 



1 

Very 

Dissatisfied 



2 

Dissatisfied 



3 

Neutral 



4 

Satisfied 



5 9 

Very No opinion/Never taken a 

Dissatisfied class in that format at UMUC 



Please rate both Face-to-Face and Online classes that you have taken. 



41. Please rate your UMUC education on each 
of the following criteria 





Face-to- 

Face 


Online 


Content of curriculum 






Quality of education 






Timeliness of feedback 






Testing and grading 






Faculty knowledge 






Faculty teaching skills 






Academic advising 






Develop critical thinking 






Rigor and scholarship 






Student faculty interaction 






Student student interaction 






Flexibility of course 






Course availability 






Relevance to career 







42. Please rate your satisfaction with UMUC’s 
provision of the following services and support 





Face-to- 

Face 


Online 


Availability of computer 
laboratories 






Availability of tutoring services 






Availability of career planning 
services 






Availability of computer 
training /information literacy 






Availability of library and other 
learning material 






Ease of registration 






Availability of technical 
assistance 






Co-op services and support 






EXCEL services and support 
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